Live plotting for LiberTEM UDFs
This example demonstrates the live plotting feature of LiberTEM that is introduced in release 0.7.0. It can update a plot with new results while LiberTEM processing is running.
The data can be downloaded at https://zenodo.org/record/5113449.
This notebook requires additional packages bqplot, bqplot_image_gl and ipywidgets. See also https://ipywidgets.readthedocs.io/en/stable/user_install.html#installing-in-classic-jupyter-notebook in case the plots don’t show.
[1]:
%matplotlib nbagg
[2]:
import os
import matplotlib.pyplot as plt
import matplotlib.colors
import ipywidgets
import IPython
import numpy as np
import libertem.api as lt
from libertem.udf import UDF
from libertem.udf.raw import PickUDF
from libertem.viz.mpl import MPLLive2DPlot
from libertem.viz.bqp import BQLive2DPlot
Plot classes
Currently, LiberTEM implements three back-ends for live plotting:
libertem.viz.mpl.MPLLive2DPlotwith amatplotlibback-end. It is the default sincematplotlibis the most popular and mature plotting package for Python. It requires about 100-200 ms to display or update a plot, which only allows a few updates per second.libertem.viz.bqp.BQLive2DPlotwith abqplot_image_glback-end. It is much faster thanmatplotliband can easily keep up with the full update rate of LiberTEM for a smoother and more responsive display.libertem.viz.gms.GMSLive2DPlotfor plotting within Digital Micrograph, Gatan Microscopy Suite. This only works using the Python scripting of recent GMS releases and is not demonstrated here.
Additional plotting back-ends can be implemented by deriving from libertem.viz.base.Live2DPlot.
Here we change the default class for live plotting to BQLive2DPlot and then change it back right away by assigning to the plot_class property of the LiberTEM Context object.
[3]:
# We set the Context to use BQLive2DPlot for live plotting
ctx = lt.Context(plot_class=BQLive2DPlot)
# We change it right back to the default MPLLive2DPlot
ctx.plot_class = MPLLive2DPlot
/home/weber/miniconda3/envs/libertem39/lib/python3.9/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 43253 instead
warnings.warn(
2023-01-12 16:25:24,233 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-zqr7hwaf', purging
2023-01-12 16:25:24,234 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-5c010dto', purging
2023-01-12 16:25:24,234 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-jnsu9o2k', purging
2023-01-12 16:25:24,234 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-2a_xuttz', purging
2023-01-12 16:25:24,234 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-ur64txas', purging
2023-01-12 16:25:24,235 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-2_u1wsmk', purging
2023-01-12 16:25:24,235 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-l_yvjlql', purging
2023-01-12 16:25:24,235 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-edtxk3wh', purging
2023-01-12 16:25:24,235 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-r7o624n2', purging
2023-01-12 16:25:24,236 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-0g279rwd', purging
2023-01-12 16:25:24,236 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-0rbn83j_', purging
2023-01-12 16:25:24,236 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-n3j0wat2', purging
2023-01-12 16:25:24,236 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-n4e61of8', purging
2023-01-12 16:25:24,237 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-e4pc8cup', purging
2023-01-12 16:25:24,237 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-ghuu0p1n', purging
2023-01-12 16:25:24,237 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space-20335/worker-ffjl_lg_', purging
2023-01-12 16:25:24,237 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CUDA", device=0)
2023-01-12 16:25:24,238 - distributed.utils - INFO - Reload module tmp3f8w1616 from .py file
2023-01-12 16:25:24,254 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=7)
2023-01-12 16:25:24,255 - distributed.utils - INFO - Reload module tmphpu43j4z from .py file
2023-01-12 16:25:24,267 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=3)
2023-01-12 16:25:24,267 - distributed.utils - INFO - Reload module tmpl7frtq47 from .py file
2023-01-12 16:25:24,275 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=16)
2023-01-12 16:25:24,275 - distributed.utils - INFO - Reload module tmpkov02hsu from .py file
2023-01-12 16:25:24,286 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=22)
2023-01-12 16:25:24,286 - distributed.utils - INFO - Reload module tmpz9kwmpcs from .py file
2023-01-12 16:25:24,315 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=13)
2023-01-12 16:25:24,315 - distributed.utils - INFO - Reload module tmp71fiolfj from .py file
2023-01-12 16:25:24,316 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=20)
2023-01-12 16:25:24,317 - distributed.utils - INFO - Reload module tmp1gkgontn from .py file
2023-01-12 16:25:24,327 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=18)
2023-01-12 16:25:24,328 - distributed.utils - INFO - Reload module tmpr5x2hgfg from .py file
2023-01-12 16:25:24,350 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=9)
2023-01-12 16:25:24,351 - distributed.utils - INFO - Reload module tmp0m8lxktt from .py file
2023-01-12 16:25:24,394 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=12)
2023-01-12 16:25:24,395 - distributed.utils - INFO - Reload module tmpgkfkifd0 from .py file
2023-01-12 16:25:24,398 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=23)
2023-01-12 16:25:24,399 - distributed.utils - INFO - Reload module tmplhj1wnfb from .py file
2023-01-12 16:25:24,410 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=5)
2023-01-12 16:25:24,410 - distributed.utils - INFO - Reload module tmpmyh60x9e from .py file
2023-01-12 16:25:24,426 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-service-0', service_id='0')
2023-01-12 16:25:24,427 - distributed.utils - INFO - Reload module tmpup6klqgy from .py file
2023-01-12 16:25:24,439 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=8)
2023-01-12 16:25:24,440 - distributed.utils - INFO - Reload module tmpu4mjstu4 from .py file
2023-01-12 16:25:24,454 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=1)
2023-01-12 16:25:24,455 - distributed.utils - INFO - Reload module tmp1iuyri_s from .py file
2023-01-12 16:25:24,477 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=15)
2023-01-12 16:25:24,478 - distributed.utils - INFO - Reload module tmppvlgvbg9 from .py file
2023-01-12 16:25:24,483 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=17)
2023-01-12 16:25:24,483 - distributed.utils - INFO - Reload module tmpv09py0t_ from .py file
2023-01-12 16:25:24,496 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=2)
2023-01-12 16:25:24,496 - distributed.utils - INFO - Reload module tmpszuwwvjv from .py file
2023-01-12 16:25:24,530 - distributed.preloading - INFO - Import preload module: /tmp/tmp3f8w1616.py
2023-01-12 16:25:24,530 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cuda-0', service_id='0')
2023-01-12 16:25:24,530 - distributed.utils - INFO - Reload module tmpaug68oyd from .py file
2023-01-12 16:25:24,531 - distributed.preloading - INFO - Import preload module: /tmp/tmpaug68oyd.py
2023-01-12 16:25:24,531 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,538 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=21)
2023-01-12 16:25:24,539 - distributed.utils - INFO - Reload module tmp6wals7j1 from .py file
2023-01-12 16:25:24,544 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=10)
2023-01-12 16:25:24,544 - distributed.utils - INFO - Reload module tmpp3rrhqm9 from .py file
2023-01-12 16:25:24,568 - distributed.preloading - INFO - Import preload module: /tmp/tmpl7frtq47.py
2023-01-12 16:25:24,568 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-3', service_id='3')
2023-01-12 16:25:24,569 - distributed.utils - INFO - Reload module tmpo_4ac6jp from .py file
2023-01-12 16:25:24,569 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=6)
2023-01-12 16:25:24,569 - distributed.preloading - INFO - Import preload module: /tmp/tmpo_4ac6jp.py
2023-01-12 16:25:24,569 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,569 - distributed.utils - INFO - Reload module tmpwz9cxz8h from .py file
2023-01-12 16:25:24,570 - distributed.preloading - INFO - Import preload module: /tmp/tmpkov02hsu.py
2023-01-12 16:25:24,570 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-16', service_id='16')
2023-01-12 16:25:24,571 - distributed.utils - INFO - Reload module tmp3qdbrznz from .py file
2023-01-12 16:25:24,571 - distributed.preloading - INFO - Import preload module: /tmp/tmp3qdbrznz.py
2023-01-12 16:25:24,571 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,571 - distributed.preloading - INFO - Import preload module: /tmp/tmphpu43j4z.py
2023-01-12 16:25:24,571 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-7', service_id='7')
2023-01-12 16:25:24,572 - distributed.utils - INFO - Reload module tmppil73x3q from .py file
2023-01-12 16:25:24,572 - distributed.preloading - INFO - Import preload module: /tmp/tmppil73x3q.py
2023-01-12 16:25:24,572 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,572 - distributed.preloading - INFO - Import preload module: /tmp/tmpz9kwmpcs.py
2023-01-12 16:25:24,572 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-22', service_id='22')
2023-01-12 16:25:24,573 - distributed.utils - INFO - Reload module tmpo59m2bon from .py file
2023-01-12 16:25:24,573 - distributed.preloading - INFO - Import preload module: /tmp/tmpo59m2bon.py
2023-01-12 16:25:24,573 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,576 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,581 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=11)
2023-01-12 16:25:24,582 - distributed.utils - INFO - Reload module tmpix_ky9sy from .py file
2023-01-12 16:25:24,594 - distributed.preloading - INFO - Import preload module: /tmp/tmp1gkgontn.py
2023-01-12 16:25:24,595 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-20', service_id='20')
2023-01-12 16:25:24,595 - distributed.utils - INFO - Reload module tmpurnf7le2 from .py file
2023-01-12 16:25:24,595 - distributed.preloading - INFO - Import preload module: /tmp/tmpurnf7le2.py
2023-01-12 16:25:24,595 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,602 - distributed.preloading - INFO - Import preload module: /tmp/tmpr5x2hgfg.py
2023-01-12 16:25:24,602 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-18', service_id='18')
2023-01-12 16:25:24,602 - distributed.preloading - INFO - Import preload module: /tmp/tmp71fiolfj.py
2023-01-12 16:25:24,602 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-13', service_id='13')
2023-01-12 16:25:24,603 - distributed.utils - INFO - Reload module tmp3t2e65w4 from .py file
2023-01-12 16:25:24,603 - distributed.utils - INFO - Reload module tmpe39iu6jb from .py file
2023-01-12 16:25:24,603 - distributed.preloading - INFO - Import preload module: /tmp/tmp3t2e65w4.py
2023-01-12 16:25:24,603 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,603 - distributed.preloading - INFO - Import preload module: /tmp/tmpe39iu6jb.py
2023-01-12 16:25:24,603 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,616 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,617 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,617 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,619 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,624 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=4)
2023-01-12 16:25:24,624 - distributed.utils - INFO - Reload module tmp4h6rbkyt from .py file
2023-01-12 16:25:24,626 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=14)
2023-01-12 16:25:24,627 - distributed.utils - INFO - Reload module tmp72q61mef from .py file
2023-01-12 16:25:24,631 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=19)
2023-01-12 16:25:24,631 - distributed.utils - INFO - Reload module tmpai7uu018 from .py file
2023-01-12 16:25:24,639 - distributed.preloading - INFO - Creating preload: from libertem.executor.dask import worker_setup; worker_setup(resource="CPU", device=0)
2023-01-12 16:25:24,639 - distributed.utils - INFO - Reload module tmpql54zi1g from .py file
2023-01-12 16:25:24,641 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,649 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,651 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,678 - distributed.preloading - INFO - Import preload module: /tmp/tmpgkfkifd0.py
2023-01-12 16:25:24,679 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-12', service_id='12')
2023-01-12 16:25:24,679 - distributed.utils - INFO - Reload module tmppfm28_kd from .py file
2023-01-12 16:25:24,679 - distributed.preloading - INFO - Import preload module: /tmp/tmppfm28_kd.py
2023-01-12 16:25:24,679 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,698 - distributed.preloading - INFO - Import preload module: /tmp/tmplhj1wnfb.py
2023-01-12 16:25:24,699 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-23', service_id='23')
2023-01-12 16:25:24,699 - distributed.utils - INFO - Reload module tmpsjbppemr from .py file
2023-01-12 16:25:24,700 - distributed.preloading - INFO - Import preload module: /tmp/tmpsjbppemr.py
2023-01-12 16:25:24,700 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,700 - distributed.preloading - INFO - Import preload module: /tmp/tmpup6klqgy.py
2023-01-12 16:25:24,700 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,706 - distributed.preloading - INFO - Import preload module: /tmp/tmpmyh60x9e.py
2023-01-12 16:25:24,706 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-5', service_id='5')
2023-01-12 16:25:24,706 - distributed.utils - INFO - Reload module tmpxxp71o_o from .py file
2023-01-12 16:25:24,707 - distributed.preloading - INFO - Import preload module: /tmp/tmpxxp71o_o.py
2023-01-12 16:25:24,707 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,710 - distributed.preloading - INFO - Import preload module: /tmp/tmp0m8lxktt.py
2023-01-12 16:25:24,710 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-9', service_id='9')
2023-01-12 16:25:24,710 - distributed.utils - INFO - Reload module tmpe0e2jzwn from .py file
2023-01-12 16:25:24,711 - distributed.preloading - INFO - Import preload module: /tmp/tmpe0e2jzwn.py
2023-01-12 16:25:24,711 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,725 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,731 - distributed.preloading - INFO - Import preload module: /tmp/tmpu4mjstu4.py
2023-01-12 16:25:24,731 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-8', service_id='8')
2023-01-12 16:25:24,732 - distributed.utils - INFO - Reload module tmpv39xr13i from .py file
2023-01-12 16:25:24,732 - distributed.preloading - INFO - Import preload module: /tmp/tmpv39xr13i.py
2023-01-12 16:25:24,732 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,744 - distributed.preloading - INFO - Import preload module: /tmp/tmppvlgvbg9.py
2023-01-12 16:25:24,744 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-15', service_id='15')
2023-01-12 16:25:24,744 - distributed.utils - INFO - Reload module tmpvlf0hedx from .py file
2023-01-12 16:25:24,744 - distributed.preloading - INFO - Import preload module: /tmp/tmpvlf0hedx.py
2023-01-12 16:25:24,744 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,746 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,746 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,752 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,756 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,765 - distributed.preloading - INFO - Import preload module: /tmp/tmpv09py0t_.py
2023-01-12 16:25:24,766 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-17', service_id='17')
2023-01-12 16:25:24,766 - distributed.utils - INFO - Reload module tmpb5_h042v from .py file
2023-01-12 16:25:24,767 - distributed.preloading - INFO - Import preload module: /tmp/tmpb5_h042v.py
2023-01-12 16:25:24,767 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,776 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,790 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,813 - distributed.preloading - INFO - Import preload module: /tmp/tmp1iuyri_s.py
2023-01-12 16:25:24,813 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-1', service_id='1')
2023-01-12 16:25:24,813 - distributed.utils - INFO - Reload module tmp717y0j27 from .py file
2023-01-12 16:25:24,814 - distributed.preloading - INFO - Import preload module: /tmp/tmp717y0j27.py
2023-01-12 16:25:24,814 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,814 - distributed.preloading - INFO - Import preload module: /tmp/tmpp3rrhqm9.py
2023-01-12 16:25:24,814 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-10', service_id='10')
2023-01-12 16:25:24,814 - distributed.utils - INFO - Reload module tmpl5r1nafi from .py file
2023-01-12 16:25:24,815 - distributed.preloading - INFO - Import preload module: /tmp/tmpl5r1nafi.py
2023-01-12 16:25:24,815 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,821 - distributed.preloading - INFO - Import preload module: /tmp/tmp6wals7j1.py
2023-01-12 16:25:24,822 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-21', service_id='21')
2023-01-12 16:25:24,822 - distributed.utils - INFO - Reload module tmp9hl4orip from .py file
2023-01-12 16:25:24,822 - distributed.preloading - INFO - Import preload module: /tmp/tmp9hl4orip.py
2023-01-12 16:25:24,822 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,829 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,858 - distributed.preloading - INFO - Import preload module: /tmp/tmpszuwwvjv.py
2023-01-12 16:25:24,858 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-2', service_id='2')
2023-01-12 16:25:24,858 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,858 - distributed.utils - INFO - Reload module tmp8_mbmg5v from .py file
2023-01-12 16:25:24,859 - distributed.preloading - INFO - Import preload module: /tmp/tmp8_mbmg5v.py
2023-01-12 16:25:24,859 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,859 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,866 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,894 - distributed.preloading - INFO - Import preload module: /tmp/tmp4h6rbkyt.py
2023-01-12 16:25:24,894 - distributed.preloading - INFO - Import preload module: /tmp/tmp72q61mef.py
2023-01-12 16:25:24,894 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-4', service_id='4')
2023-01-12 16:25:24,894 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-14', service_id='14')
2023-01-12 16:25:24,894 - distributed.utils - INFO - Reload module tmpp1axw22y from .py file
2023-01-12 16:25:24,894 - distributed.utils - INFO - Reload module tmpa7y_ti4x from .py file
2023-01-12 16:25:24,895 - distributed.preloading - INFO - Import preload module: /tmp/tmpp1axw22y.py
2023-01-12 16:25:24,895 - distributed.preloading - INFO - Import preload module: /tmp/tmpa7y_ti4x.py
2023-01-12 16:25:24,895 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,895 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,901 - distributed.preloading - INFO - Import preload module: /tmp/tmpai7uu018.py
2023-01-12 16:25:24,901 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-19', service_id='19')
2023-01-12 16:25:24,901 - distributed.utils - INFO - Reload module tmpoynbj8h2 from .py file
2023-01-12 16:25:24,902 - distributed.preloading - INFO - Import preload module: /tmp/tmpoynbj8h2.py
2023-01-12 16:25:24,902 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,903 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,906 - distributed.preloading - INFO - Import preload module: /tmp/tmpwz9cxz8h.py
2023-01-12 16:25:24,906 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-6', service_id='6')
2023-01-12 16:25:24,907 - distributed.utils - INFO - Reload module tmpdgqjvtv2 from .py file
2023-01-12 16:25:24,907 - distributed.preloading - INFO - Import preload module: /tmp/tmpdgqjvtv2.py
2023-01-12 16:25:24,907 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,909 - distributed.preloading - INFO - Import preload module: /tmp/tmpql54zi1g.py
2023-01-12 16:25:24,909 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-0', service_id='0')
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Import preload module: /tmp/tmpix_ky9sy.py
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Creating preload: from libertem.common.tracing import maybe_setup_tracing; maybe_setup_tracing(service_name='default-cpu-11', service_id='11')
2023-01-12 16:25:24,910 - distributed.utils - INFO - Reload module tmpp3uwrt5g from .py file
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Import preload module: /tmp/tmpp3uwrt5g.py
2023-01-12 16:25:24,910 - distributed.utils - INFO - Reload module tmpgz7yxmln from .py file
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,910 - distributed.preloading - INFO - Import preload module: /tmp/tmpgz7yxmln.py
2023-01-12 16:25:24,911 - distributed.preloading - INFO - Creating preload: libertem.preload
2023-01-12 16:25:24,938 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,939 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,945 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,951 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,954 - distributed.preloading - INFO - Import preload module: libertem.preload
2023-01-12 16:25:24,955 - distributed.preloading - INFO - Import preload module: libertem.preload
[4]:
data_base_path = os.environ.get("TESTDATA_BASE_PATH", "/home/alex/Data/")
ds = ctx.load("auto", path=os.path.join(data_base_path, "20200518 165148/default.hdr"))
# This internal method is used here to artificially create more partitions than necessary
# for better demonstration of live plotting on machines with few cores.
# This reduces performance, should NOT be used in production and can change without
# notice in future releases.
ds.set_num_cores(32)
2023-01-12 16:25:25,468 - distributed.worker - WARNING - Compute Failed
Key: _do_detect-4a5d47dc-598f-491d-b7f0-04e219a5ed91
Function: _do_detect
args: ()
kwargs: {}
Exception: 'DataSetException("OSError(\'Unable to open file (file signature not found)\')")'
Demo UDFs
We implement three simple user-defined functions (UDFs) to demonstrate the various features and options for live plotting. They calculate a map of the navigation space with sum and maximum, a map of the signal space with sum and maximum, and a global maximum.
See https://libertem.github.io/LiberTEM/udf.html for more information on UDFs.
[5]:
class DemoNavUDF(UDF):
def get_result_buffers(self):
return {
'nav_sum': self.buffer(kind='nav', dtype='float32'),
'nav_max': self.buffer(kind='nav', dtype='float32'),
}
def process_frame(self, frame):
self.results.nav_sum[:] = np.sum(frame)
self.results.nav_max[:] = np.max(frame)
class DemoSigUDF(UDF):
def get_result_buffers(self):
return {
'sig_sum': self.buffer(kind='sig', dtype='float32'),
'sig_max': self.buffer(kind='sig', dtype='float32'),
}
def process_frame(self, frame):
self.results.sig_sum += frame
np.maximum(self.results.sig_max, frame, out=self.results.sig_max)
def merge(self, dest, src):
dest.sig_sum += src.sig_sum
np.maximum(dest.sig_max, src.sig_max, out=dest.sig_max)
class DemoSingleUDF(UDF):
def get_result_buffers(self):
return {
'maximum': self.buffer(kind='single', dtype='float32'),
}
def process_frame(self, frame):
self.results.maximum[:] = np.maximum(self.results.maximum, np.max(frame))
def merge(self, dest, src):
dest.maximum[:] = np.maximum(dest.maximum, src.maximum)
The UDFs are instantiated and stored in a list for subsequent use.
[6]:
udfs = [DemoNavUDF(), DemoSigUDF(), DemoSingleUDF()]
Plot all plottable channels
Note how Context.run_udf() can execute several UDFs in one pass since release 0.7.0 by passing a list or tuple of UDFs.
By setting plots=True you can plot all channels that have a 2D shape after applying np.squeeze. The DemoSingleUDF is not plotted since its only channel maximum is a single value. This triggers a warning.
[7]:
res = ctx.run_udf(dataset=ds, udf=udfs, plots=True)
/home/weber/LiberTEM/LiberTEM/src/libertem/api.py:1212: UserWarning: No plottable channels found for UDF #2: DemoSingleUDF, not plotting.
warnings.warn(
Select specific channels
We can specify which channels should be plotted by passing a nested list with channel names for each of the UDFs as the plots parameter.
[8]:
res = ctx.run_udf(dataset=ds, udf=udfs, plots=[['nav_sum'], ['sig_sum', 'sig_max']])
Apply a function to a channel
Together with the channel name we can specify a function to apply to the channel before plotting. This can plot a channel that doesn’t work with default plotting by transforming it into something plottable. Here we demonstrate this with the single-valued maximum channel.
The same channel can be plotted several times with a different function. This is particularly useful to plot the absolute value and the phase angle for complex numbers: ('chan', np.abs), ('chan', np.angle). Here we demonstrate this with thresholding.
[9]:
# Just some thresholding for demonstration
def larger(x):
return x > 9.3e4
def smaller(x):
return x < 9e4
[10]:
plot_list=[
['nav_sum', ('nav_sum', larger), ('nav_sum', smaller)],
[],
[('maximum', lambda x: x.reshape((1, 1)))]
]
res = ctx.run_udf(dataset=ds, udf=udfs, plots=plot_list)
Custom plot setup
If we want to have more control over the plots and their layout, we can instantiate plots separately and pass them to run_udf() via the plots parameter. This also allows to set additional parameters such as the title. Note how MPLLive2DPlot passes additional keyword arguments, in this case the color map cmap and the norm, to matplotlib.pyplot.imshow.
The UDF passed to the live plot via the udf parameter has to be the same instance that is later run in run_udf() to ensure that results are assigned to the correct plot.
[11]:
live_plot = MPLLive2DPlot(
dataset=ds,
udf=udfs[1],
# We add 1 for log scale display since there are zeros in the result
channel=('sig_sum', lambda x: 1 + x),
title="1 + x, log scaled display",
# These parameters are passed to imshow()
cmap='inferno',
norm=matplotlib.colors.LogNorm()
)
[12]:
live_plot.display()
# We can access and modify the matplotlib objects, in this case to add a color bar
live_plot.fig.colorbar(live_plot.im_obj)
[12]:
<matplotlib.colorbar.Colorbar at 0x7efdb870c0a0>
[13]:
# Run the UDF, showing live results in the plot above
res = ctx.run_udf(dataset=ds, udf=udfs, plots=[live_plot])
Gridded display
Here we use bqplot and arrange the plots in a grid. The same method can also be used with MPLLive2DPlot if the output method of matplotlib is switched to use ipywidgets. Just install ipympl, replace %matplotlib nbagg at the top of the notebook with %matplotlib widget and restart the kernel.
Note that at the time of writing this notebook, a MPLLive2DPlot combined with display method “widget” will only update live if it is created and displayed separately, not if this is done automatically in run_udf() .
[14]:
# NBVAL_IGNORE_OUTPUT
# (output is ignored in nbval run because it somehow doesn't play nice with bqplot)
live_plots = [
BQLive2DPlot(dataset=ds, udf=udfs[0], channel='nav_sum'),
BQLive2DPlot(dataset=ds, udf=udfs[0], channel=('nav_sum', larger)),
BQLive2DPlot(dataset=ds, udf=udfs[0], channel=('nav_sum', smaller)),
BQLive2DPlot(dataset=ds, udf=udfs[1], channel='sig_sum'),
BQLive2DPlot(dataset=ds, udf=udfs[1], channel='sig_max'),
BQLive2DPlot(dataset=ds, udf=udfs[2], channel=('maximum', lambda x: x.reshape((1, 1)))),
]
outputs = []
for p in live_plots:
# We capture the display of the widget with ipywidgets.Output() to arrange it later
output = ipywidgets.Output()
with output:
p.display()
# Some bqplot-specific tweaks to reduce white space between plots
p.figure.fig_margin={'top': 50, 'bottom': 0, 'left': 25, 'right': 25}
p.figure.layout.height = '300px'
outputs.append(output)
[15]:
# Display the captured outputs in a grid
ipywidgets.VBox([
ipywidgets.HBox(outputs[:3]),
ipywidgets.HBox(outputs[3:]),
])
[15]:
What it looks like
The gridded layout is currently not preserved when saving the notebook. This PNG shows a screenshot:
[16]:
res = ctx.run_udf(dataset=ds, udf=udfs, plots=live_plots)
User-defined plotting
In some cases one may want to plot results that are based on combining several channels of a UDF. As a specific example, when strain mapping with https://libertem.github.io/LiberTEM-blobfinder/ one may want to plot the angle between the a and b vectors in the fit result. Live2DPlot allows to pass a callable as the channel argument that will receive a complete UDF result together with a damage buffer that indicates which part of the navigation space has already been processed.
The callable is expected to return an ndarray together with damage that indicates the positions in the result that hold valid data. If the result is derived from kind='nav' buffers, the damage parameter of the function can be passed through. If the result is derived from other buffers, i.e. the entire result array contains valid data, the function can just return True for the damage.
Here we will plot the ratio between sum and max for demonstration purposes.
[17]:
def nav_ratio(udf_result, damage):
# We use the damage to exclude invalid data
# Avoid divide by 0
where = damage & (np.abs(udf_result['nav_max'].data) > 1e-12)
result = np.divide(
udf_result['nav_sum'].data,
udf_result['nav_max'].data,
where=where
)
return result, damage
def sig_ratio(udf_result, damage):
# Avoid divide by 0
where = np.abs(udf_result['sig_max'].data) > 1e-12
result = np.divide(
udf_result['sig_sum'].data,
udf_result['sig_max'].data,
where=where
)
return result, True
[18]:
live_plots = [
MPLLive2DPlot(dataset=ds, udf=udfs[0], channel=nav_ratio),
MPLLive2DPlot(dataset=ds, udf=udfs[1], channel=sig_ratio),
]
for p in live_plots:
p.display()
p.fig.colorbar(p.im_obj)
[19]:
res = ctx.run_udf(dataset=ds, udf=udfs, plots=live_plots)
Region of interest (ROI)
The shape of result buffers can depend on the ROI in some cases. The most notable example is libertem.udf.raw.PickUDF. For that reason we should pass the ROI to the constructor of Live2DPlot subclasses if the UDFs will be run with a ROI.
[20]:
roi = np.zeros(ds.shape.nav, dtype=bool)
roi[0,0] = True
pick_udf = PickUDF()
[21]:
# This just works, the ROI is handled correctly internally
ctx.run_udf(dataset=ds, udf=pick_udf, roi=roi, plots=True)
[21]:
{'intensity': <BufferWrapper kind=single dtype=uint8 extra_shape=(1, 256, 256)>}
[22]:
# Here we have to pass the ROI
# If this was omitted, the PickUDF would allocate memory for the entire dataset and the plot would try to display that!
live_plot_pick = MPLLive2DPlot(dataset=ds, udf=pick_udf, roi=roi, channel='intensity')
live_plot_pick.display()
[23]:
ctx.run_udf(dataset=ds, udf=pick_udf, roi=roi, plots=[live_plot_pick])
[23]:
{'intensity': <BufferWrapper kind=single dtype=uint8 extra_shape=(1, 256, 256)>}
Note that the first axis of the result of PickUDF matches the number of entries in the ROI that are True. If this is a single one, this axis is “squeezed out” with default plotting. If functions are used to transform the result, this is the responsibility of the user. If more than one entry in the ROI is True, the result can’t be plotted with default methods since it can’t be squeezed to 2D anymore.
[24]:
# NBVAL_RAISES_EXCEPTION
# Error, wrong shape!
ctx.run_udf(dataset=ds, udf=pick_udf, roi=roi, plots=[[('intensity', lambda x: x)]])
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[24], line 3
1 # NBVAL_RAISES_EXCEPTION
2 # Error, wrong shape!
----> 3 ctx.run_udf(dataset=ds, udf=pick_udf, roi=roi, plots=[[('intensity', lambda x: x)]])
File ~/LiberTEM/LiberTEM/src/libertem/api.py:775, in Context.run_udf(self, dataset, udf, roi, corrections, progress, backends, plots, sync)
773 with tracer.start_as_current_span("Context.run_udf"):
774 if sync:
--> 775 return self._run_sync(
776 dataset=dataset,
777 udf=udf,
778 roi=roi,
779 corrections=corrections,
780 progress=progress,
781 backends=backends,
782 plots=plots,
783 iterate=False,
784 )
785 else:
786 return self._run_async(
787 dataset=dataset,
788 udf=udf,
(...)
794 iterate=False,
795 )
File ~/LiberTEM/LiberTEM/src/libertem/api.py:1013, in Context._run_sync(self, dataset, udf, roi, corrections, progress, backends, plots, iterate)
1011 if enable_plotting:
1012 with tracer.start_as_current_span("prepare_plots"):
-> 1013 plots = self._prepare_plots(udfs, dataset, roi, plots)
1015 if corrections is None:
1016 corrections = dataset.get_correction_data()
File ~/LiberTEM/LiberTEM/src/libertem/api.py:1239, in Context._prepare_plots(self, udfs, dataset, roi, plots)
1227 for channel in udf_channels:
1228 p0 = self.plot_class(
1229 dataset,
1230 udf=udf,
(...)
1237 ),
1238 )
-> 1239 p0.display()
1240 plots.append(p0)
1241 return plots
File ~/LiberTEM/LiberTEM/src/libertem/viz/mpl.py:75, in MPLLive2DPlot.display(self)
73 def display(self):
74 self.fig, self.axes = plt.subplots()
---> 75 self.im_obj = self.axes.imshow(self.data, **self.kwargs)
76 # Set values compatible with log norm
77 self.im_obj.norm.vmin = 1
File ~/miniconda3/envs/libertem39/lib/python3.9/site-packages/matplotlib/_api/deprecation.py:454, in make_keyword_only.<locals>.wrapper(*args, **kwargs)
448 if len(args) > name_idx:
449 warn_deprecated(
450 since, message="Passing the %(name)s %(obj_type)s "
451 "positionally is deprecated since Matplotlib %(since)s; the "
452 "parameter will become keyword-only %(removal)s.",
453 name=name, obj_type=f"parameter of {func.__name__}()")
--> 454 return func(*args, **kwargs)
File ~/miniconda3/envs/libertem39/lib/python3.9/site-packages/matplotlib/__init__.py:1433, in _preprocess_data.<locals>.inner(ax, data, *args, **kwargs)
1430 @functools.wraps(func)
1431 def inner(ax, *args, data=None, **kwargs):
1432 if data is None:
-> 1433 return func(ax, *map(sanitize_sequence, args), **kwargs)
1435 bound = new_sig.bind(ax, *args, **kwargs)
1436 auto_label = (bound.arguments.get(label_namer)
1437 or bound.kwargs.get(label_namer))
File ~/miniconda3/envs/libertem39/lib/python3.9/site-packages/matplotlib/axes/_axes.py:5610, in Axes.imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, interpolation_stage, filternorm, filterrad, resample, url, **kwargs)
5602 self.set_aspect(aspect)
5603 im = mimage.AxesImage(self, cmap=cmap, norm=norm,
5604 interpolation=interpolation, origin=origin,
5605 extent=extent, filternorm=filternorm,
5606 filterrad=filterrad, resample=resample,
5607 interpolation_stage=interpolation_stage,
5608 **kwargs)
-> 5610 im.set_data(X)
5611 im.set_alpha(alpha)
5612 if im.get_clip_path() is None:
5613 # image does not already have clipping set, clip to axes patch
File ~/miniconda3/envs/libertem39/lib/python3.9/site-packages/matplotlib/image.py:710, in _ImageBase.set_data(self, A)
706 self._A = self._A[:, :, 0]
708 if not (self._A.ndim == 2
709 or self._A.ndim == 3 and self._A.shape[-1] in [3, 4]):
--> 710 raise TypeError("Invalid shape {} for image data"
711 .format(self._A.shape))
713 if self._A.ndim == 3:
714 # If the input data has values outside the valid range (after
715 # normalisation), we issue a warning and then clip X to the bounds
716 # - otherwise casting wraps extreme values, hiding outliers and
717 # making reliable interpretation impossible.
718 high = 255 if np.issubdtype(self._A.dtype, np.integer) else 1
TypeError: Invalid shape (1, 256, 256) for image data
[25]:
# Works, we squeeze the extra axis out
ctx.run_udf(dataset=ds, udf=pick_udf, roi=roi, plots=[[('intensity', lambda x: x.squeeze())]])
[25]:
{'intensity': <BufferWrapper kind=single dtype=uint8 extra_shape=(1, 256, 256)>}
[26]:
# We have a ROI with two entries
roi[0, 0] = True
roi[-1, -1] = True
[27]:
# No plot shown since the result is 3D now
ctx.run_udf(dataset=ds, udf=pick_udf, roi=roi, plots=True)
/home/weber/LiberTEM/LiberTEM/src/libertem/api.py:1212: UserWarning: No plottable channels found for UDF #0: PickUDF, not plotting.
warnings.warn(
[27]:
{'intensity': <BufferWrapper kind=single dtype=uint8 extra_shape=(2, 256, 256)>}
[ ]: